REF: pytables do string conversion early to set attributes in fewer places #30058
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n-pytables-maybe_convert
…n-pytables-maybe_convert
…n-pytables-maybe_convert
…n-pytables-maybe_convert
…n-pytables-maybe_convert
jreback
reviewed
Dec 4, 2019
| @@ -4842,6 +4774,74 @@ def _unconvert_index(data, kind: str, encoding=None, errors="strict"): | |||
| return index | |||
|
|
|||
|
|
|||
| def _maybe_convert_for_string_atom( | |||
| name: str, block, existing_col, min_itemsize, nan_rep, encoding, errors | |||
There was a problem hiding this comment.
just cut / paste?
add a doc-string / types at some point
There was a problem hiding this comment.
Mostly cut/paste, but from a couple different places instead of one big chunk. The not block.is_object check on 4782 is new, wasnt needed before because before we were doing things in a different order.
The only real change is the reshape on 4827 and assertion on 4828 are new.
| data = block.values | ||
|
|
||
| # see if we have a valid string type | ||
| inferred_type = lib.infer_dtype(data.ravel(), skipna=False) |
There was a problem hiding this comment.
don't you already have this from above?
There was a problem hiding this comment.
no, because we did a fillna on 4802
|
thanks, that does simplify the nested logic a bit |
|
i figured out last night it allows further simplfications even more than i expected. coming up before long. |
proost
pushed a commit
to proost/pandas
that referenced
this pull request
Dec 19, 2019
proost
pushed a commit
to proost/pandas
that referenced
this pull request
Dec 19, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Next-step following #29979.
A lot of work is currently done in set_atom_string (a self-mutating method) that can be done in an earlier, not-mutating call. By making that move, we simplify set_atom quite a bit, and get closer to getting all the attribute-setting into the constructor